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(54) Local positioning apparatus, and method therefor 



(57) An apparatus and method for correctly deter- 
mining the position of a vehicle in a traffic lane by ob- 
taining correct informatbn about the position of the traf- 
fic lane without being affected by variations In the road 
surface, weather, time of day, or such imaging condi- 
tions as fixed or moving lighting, are provided. An edge 
signal (SH) of a high spatial frequency component and 
a luminance signal (SL) of a low spatial frequency com- 
ponent of a digital image signal (Si) representing the 
view of the local area (Vi) to the front of a vehicle are 
extracted. A road contour signal (Sre) is then extracted 
from the edge signal (SH). and a road region signal (Srr) 
is extracted from the luminance signal (SL). The position 
of the lane Sre is then detected with high precision by 
evaluating the lane contour Sre based on the road re- 
gion signal Srr and lane contour data Sre, 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 5 

The present invention relates to a local positioning 
apparatus used in a local positioning system particularly 
suited to detecting the condition of a subject, such as 
an automobile or other motor vehicle, on a road while fO 
either stationary or moving based on such local posi- 
tioning information as the relative location, velocity, and 
attitude of the subject within a localized region with ref- 
erence to an image obtained by imaging the area in front 
of said subject. More specifically, the present invention is 
relates to a local positioning apparatus for correctly de- 
tecting the position of a subject within a localized region 
without being affected by changes in the condition of the 
road surface, weather, lime of day, fixed lighting, moving 
lighting, and other changes in the imaging condition. 20 

A local positioning apparatus according to the prior 
art used in an automobile is shown in Fig. 25 and de- 
scribed below. As shown in Fig. 25, this conventional 
local positioning apparatus LPP comprises an edge ex- 
tractor 1 P, threshold generator 3P, contour extractor 5P, 25 
matching operator 9P, lane (marker) contour extractor 
11 P. region limiler 13P, current position detector 15P, 
curvature detector 17P, and yaw angle detector 19P. 

The edge extractor 1 P is connected to a digital im- 
aging apparatus 100 (Fig. 1 ). The digital imaging appa- 30 
ratus 100 is mounted to the subject, which in this expla- 
nation of the prior art and the specification of th^ present 
invention below is. by way of example only, an automo- 
bile AM (Fig. 4), for capturing a perspective image Vi of 
the view to the front of the automobile AM in the direction 55 
of travel, and generating a digital image signal Si of the 
perspective image Vi. Included in the perspective image 
Vi are an image of the road, the lane markers Lm1 and 
Lm2 defining the boundaries (sides) of the lane Lm in 
which the automobile AM is currently travelling, and a 40 
lane marker Lm3 defining the far boundary of an adja- 
cent lane (Fig. 5). The edge extractor 1 P extracts the 
edge pixels of the lane markers Lml . Lm2, and LmS 
from the digital image signal Si. and generates an ex- 
tracted edge pixel signal Sx'. The extracted edge pixel 45 
signal Sx' contains only the edge pixels extracted by the 
edge extractor IP, and thus represents an extracted 
edge image Vx'. 

Using a known method, the threshold generator 3P 
scans the extracted edge pixel signal Sx' to extract a so 
line for each of the lanes Lm delineated by the lane 
markers Lm1 , Lm2, and Lm3. and determines a thresh- 
old value Eth' for extracting pixels representing the con- 
tours of the lane markings from the extracted edge pixel 
signal Sx'. Using the supplied threshold value Eth', the ss 
contour extractor 5P then extracts contour pixels and 
generates an extracted contour signal Sc' representing 
the contour lines of the lane markings. 



The matching operator 9P then determines the line 
segment or arc matching the contour lines contained in 
the extracted contour signal Sc'. and generates match- 
ing data Sm* containing all of the matching line and arc 
segments. 

The lane contour extractor 11 P then compares the 
matching data Sm' with typical lane dimension charac- 
teristics stored in memory to extract the matching line 
elements meeting these dimensional criteria as the con- 
tour lines of the lane, and outputs the result as lane ex- 
traction signal Smc'. 

Based on this lane extraction signal Smc', the re- 
gion limiter 13P defines a certain region around the ex- 
tracted lane, and generates a region signal Sr' delimiting 
this lane region. By feeding this region signal Sr' back 
to the edge extractor 1 P. the edge extractor 1 P limits the 
area within the perspective image Vi used for edge ex- 
traction to the region limits defined by the region limiter 
13R 

Using the lane extraction signal Smc' from the lane 
contour extractor IIP, the current position detector 15P 
detects the position of the automobile AM on the road, 
or more specifically in relationship to the lane being fol- 
lowed. 

The curvature detector 17P detects the curvature 
of the lane being followed while the automobile AM is 
moving. The yaw angle detector 19P detects the angle 
of the automobile AM relative to the lane, i.e., whether 
the automobile AM is travelling parallel to the sides of 
the lane or is following a course while would result in the 
automobile AM leaving the current lane being followed. 

It should be noted that all of the processes de- 
scribed above are based on the perspective image Vi of 
the area to the front of the vehicle obtained by the digital 
imaging apparatus 100. It is obvious, however, that cor- 
rect information about the lane dimensions cannot be 
obtained from the perspective image Vi because the 
perspective image is a simple two-dimensional repre- 
sentation of three-dimensional space. Specifically, 
shapes change in the perspective image Vi as the dis- 
tance from the digital imaging apparatus 100 increases 
with the size of an object at a distance from the digital 
imaging apparatus 1 00 being displayed smallerthan the 
same object in close proximity to the digital imaging ap- 
paratus 100. In addition, the edges of a road or lane are 
indistinct in a perspective image Vi. making edge detec- 
tion difficult. 

Road conditions are also not constant, and this fur- 
ther complicates road edge detection. For example, rec- 
ognizing the contour of a road or lane by means of edge 
detection is, in fact, impossible when the side of a road 
or a lane marker is hidden by such as vegetation, dirt, 
or gravel. Edge detection is also not practically possible 
when the lane markers are not recognizable in part or 
in full because of soiling, damage, or other cause. 

The brightness of the road surface is also extremely 
variable, and is affected by such factors as the weather, 
whether the road is inside a tunnel, and whether it is day 
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or night. Stationary lights installed in tunnels and beside 
the road also only partially and locally illuminate the road 
surface, and spots of extreme brightness or darkness 
can result even from the same road surface. These con- 
ditions are further complicated at night by irregular s 
changes in illumination resulting both from the head- 
lights of the subject automobile Af^ and the headlights 
of other vehicles. Accurately determining the edge de- 
tection threshold value Eth' is effectively impossible un- 
der environments subject to changes in driving condi- io 
tions. time of day, and weather, as well as dynamic 
changes in brightness in the perspective image Vi due 
to fixed or moving lighting even when the driving condi- 
tions, time, and weather remain constant. 

In other words, it is not possible to obtain accurate '5 
dimensional information about the road and lane from 
an extracted contour signal Sc" that is based on such an 
inaccurate threshold value Eth*. 

It is therefore clearly extremely dangerous to detect 
the local positioning of a vehicle relalive to a road or lane 
based on such unreliable, inaccurate, and distorted di- 
mensional information, and to detect the road curvature 
and vehicular yaw based on such erroneously deter- 
mined positioning information. 

SUMMARY OF THE INVENTION 

The object of the present invention is therefore to 
resolve the above problems by providing a local posi- 
tioning apparatus for detecting the local position of a 30 
subject capable of advancing in a direction related to a 
lane within a localized region based on a digital image 
signal representing a localized region in the direction of 
subject travel. To achieve said object, said local posi- 
tioning apparatus comprises a first image signal gener- 35 
ating means for extracting a high frequency component 
of a spatial frequency from said digital image signal to 
generate an edge signal; a contour extracting means for 
extracting lane contours based on the edge signal, and 
generating lane contour data; a second image signal -to 
generating means for extracting a low frequency com- 
ponent of a spatial frequency from said digital image sig- 
nal to generate a luminance signal; a lane area extract- 
ing means for extracting the lane area based on the lu- 
minance signal, and generating lane area data; and a -^5 
lane detecting means for detecting a lane position and 
generating a lane detection signal based on the lane 
contour data and lane area data. 

BRIEF DESCRIPTION OF THE DRAWINGS so 

These and other objects and features of the present 
invention will become clear from the following descrip- 
tion taken in conjunction with the preferred embodi- 
ments thereof with reference to the accompanying ss 
drawings throughout which like parts are designated by 
like reference numerals, and in which: 



Fig. 1 is a block diagram of a local positioning ap- 
paratus according to a first embodiment of the 
present invention. 

Fig. 2 is a detailed block diagram of essential com- 
ponents of a local positioning apparatus shown in 
Fig. 1. 

Fig. 3 is used to describe the coordinate system of 
a perspective image in the present invention, 
Fig. 4 is used to describe the coordinate system of 
a birds'-eye image in the present invention, 
Fig. 5 is a perspective image of the view to the front 
of the automobile AM that is processed by the local 
positioning apparatus of the invention. 
Fig. 6 shows the edges extracted from the perspec- 
tive image in Fig. 5. 

Fig. 7 is used to described a method of extracting 
contours from the edge image in Fig. 5, 
Fig. 8 shows the contour lines extracted from the 
edge image in Fig. 5, 

Fig. 9 is used to describe how the processing region 
of the perspective image is limited by the contour 
extraction region limiter in Fig. 2, 
Fig. 10 is used to describe a perspective image after 
region limited by the method shown in 9, 
Fig. 11 is used to describe the luminance image ex- 
tracted from the front perspective image shown in 
Fig. 5. 

Fig. 12 is a graph of the relationship between pixel 
luminance in the luminance image in Fig. 1 1 and the 
pixel count having a particular luminance value, 
Fig. 1 3 is a graph of experimental values showing 
the relationship between the luminance values of 
pixels in the road area of the luminance image, and 
the focal distance of the road surface from which 
the corresponding pixel was captured, 
Fig. 1 4 is used to describe the road brightness set- 
ling method of the road brightness setter shown in 
Fig. 2. 

Fig. 15 is used to describe the road area extraction 
method of the road image extractor in Fig. 2, 
Fig. 16 is used to describe a method of extracting 
the road area edges from the road area pixels ex- 
tracted as shown in Fig. 1 5. 

Fig. 17 shows the road image after linear approxi- 
mation using the method shown in Fig. 16, 
Fig. 18 is used to describe the operation of the road 
region setter. 

Fig. 1 9 shows a birds'-eye image after coordinate 
conversion of the road contour signal by the coor- 
dinate converter in Fig. 2, 

Fig. 20 is used to describe the curve patterns ob- 
tained by Hough conversion of the birds'-eye con- 
tour signal Sec in Fig 19. 

Fig. 21 is an alternative example of the curve pat- 
terns obtained by Hough conversion of the birds'- 
eye contour signal Sec in Fig. 19, 
Fig. 22 is an alternative example of the curve pat- 
terns obtained by Hough conversion of the birds'- 
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eye contour signal Sec in Fig. ig, 

Fig. -23 is used to describe in-lane position detection 

by the in-lane position detector. 

Fig. 24 is a flow chart used to describe the overall 

operatbn of the local positioning apparatus shown 5 

In Fig. 1, 

Fig. 25 Is a block diagram of a local positioning ap- 
paratus according to the prior art, 
Fig. 26 is a block diagram of a road area extraction 
apparatus according to a second embodiment of 
the present Invention, 

Fig. 27 Is an example of a brightness histogram 
generated from the road image by the brightness 
histogram generator in Fig. 26, 

Fig. 28 is a flow chart used to describe the overall is 
operation of the road area extraction apparatus 
shown in Fig. 26, 

Fig. 29 is used to describe the method of detection 
lane edges from the extracted lane area by means 
of the road area extraction apparatus shown in Fig. 20 
26. 

Fig. 30 is a block diagram of a road area extraction 
apparatus according to a first alternative of second 
embodiment of the present invention. 
Fig. 31 is used to describe the area occupied by the 25 
lane area in the brightness histogram generated by 
the road area extraction apparatus in Fig. 30, 
Fig. 32 is used to describe the brightness distribu- 
tion obtained by segmenting the road image by 
means of the lane Image separator, and the bright- 30 
ness distribution pattern obtained by the brightness 
distribution pattern detector in Fig. 30. 
Fig. 33 is a flow chart used to describe the overall 
operatbn of the road area extraction apparatus 
shown in Fig. 30, 3S 
Fig. 34 is a block diagram of a road area extraction 
apparatus according to a second alternative of the 
second embodiment of the present invention, 
Fig. 35 is an example of a brightness histogram 
generated from the road image by the brightness 40 
histogram generator in Fig. 34, 
Fig. 36 Is a flow chart used to describe the overall 
operatbn of the road area extraction apparatus 
shown in Fig. 34, 

Fig. 37 is a block diagram of a road area extraction 45 
apparatus according to a third alternative of the sec- 
ond embodiment of the present invention, 
Fig. 38 is an example of a brightness histogram 
generated from the road image by the brightness 
histogram generator in Fig. 37, so 
Fig. 39 is used to describe road image segmenta- 
tion by the image separator 43 shown In Fig. 37, 
Fig. 40 is the first part of a flow chart used to de- 
scribe the overall operation of the road area extrac- 
tion apparatus shown in Fig. 37, ss 
Fig. 41 is the second part of a flow chart used to 
describe the overall operation of the road area ex- 
traction apparatus shown in Fig. 37. 



Fig. 42 is a block diagram of a road area extraction 
apparatus according to a fourth alternative of the 
second embodiment of the present invention, 
Fig. 43 is used to describe the relationship between 
extracted lane area obtained by the road area ex- 
traction apparatus shown in Fig. 42, and a triangle 
formed in the image from the vanishing point and 
road edge pixels, 

Fig. 44 Is a flow chart used to describe the overall 
operation of the road area extraction apparatus 
shown in Fig. 42. and 

Fig. 45 is a block diagram of a road area extraction 
apparatus according to the prior art. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

Flg. l is a block diagram of a local positioning ap- 
paratus LP according to a first embodiment of the 
present invention. This local positioning apparatus LP 
is used on board motor vehicles such as automobiles 
AhA and other types of vehicles that are capable of tra- 
versing a freely defined path of travel. 

As shown in Fig. 1, the local positioning apparatus 
LP comprises a digital imaging apparatus 100, spatial 
frequency separator 200, lane area detector 300, lane 
contour detector 400, lane detector 500, and electronic 
control unit (ECU) 700. Note that the ECU 700 is a de- 
vice commonly used and known in the automobile in- 
dustry, and is used to detect the vehicle condition as rep- 
resented by the speed of travel and steering condition, 
generate a vehicle condition signal Sc. which includes 
a velocity signal Svand steering signal Ss. and controls 
the varbus electrical devices of the vehicle. 

The digital Imaging apparatus 100 comprises a Ph 
X PV pixel matrix of imaging elements, and continuously 
captures Images of the subject, i.e., the area to the front 
in the direction of travel, where Ph is the number of hor- 
izontal pixels and PV is the number of vertical pixels. 
Increasing pixel counts Ph and PV can be used to im- 
prove Image resolution, and to therefore improve the po- 
sitioning precision of the present invention. Increasing 
the pixel count, however, also increases the manufac- 
turing cost, and the pixel count must therefore be deter- 
mined by balancing the required resolution against the 
manufacturing cost. While a digital imaging apparatus 
100 according to an exemplary embodiment of the in- 
vention described below has a 428 (Ph) by 268 (PV) pix- 
el imaging device, the invention shall obviously not be 
so limited. 

The digital imaging apparatus 100 Is mounted to- 
ward the front of the automobile AM, and from this po- 
sition continuously images a view Vi in the direction of 
travel of the automobile AM to generate a digital image 
signal Si. This view Vi is a perspective image Vi of the 
area to the front of the automobile AM as seen from the 
automobile AM. Furthermore, this perspective image Vi 
can be either a still image or a motion picture. 
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The spatial frequency separator 200 is connected 
to the digital imaging apparatus 1 00, and receives there- 
from the digital image signal SI. The spatial frequency 
separator 200 thus extracts a low frequency component 
from the digital image signal Si to generate a low spatial s 
frequency signal SL, and extracts a high frequency com- 
ponent to generate a high spatial frequency signal SH. 

The lane area detector 300 is connected to the spa- 
tial frequency separator 200. and receives therefrom the 
low spatial frequency signal SL, based on which the lane io 
area detector 300 detects the road surface from the per- 
spective image Vi to generate a road region signal Srr. 

The lane contour detector 400 is likewise connected 
to the spatial frequency separator 200. receives there- 
from the high spatial frequency signal SH, thus detects 
the contour ot the road from the perspective image Vi. 
and generates a road contour signal Sre. The lane con- 
tour detector 400 also generates a region limiting signal 
Sr that limits the scanning area used for contour detec- 
tion, and inputs this signal to the spatial frequency sep- 20 
arator 200. More specifically, the region limiting signal 
Sr is used to limit the area in the perspective image Vi 
that is scanned for contour detection to the area around 
the detected contour as a means of reducing the 
processing load. The spatial frequency separator 200 2S 
then extracts the high spatial frequency signal SH only 
from the area of the digital image signal Si specified by 
the region limiting signal Sr. 

The lane detector 500 is connected to the lane area 
detector 300 and the lane contour detector 400, and 30 
thus receives both the road region signal Srr and the 
road contour signal Sre. Based on the road region signal 
Srr. the lane detector 500 determines whether the road 
contour indicated by the road contour signal Sre is cor- 
rect, and generates a road judgement signal Sj indicat- 35 
ing the result of this evaluation. The road judgement sig- 
nal Sj is then fed back to the lane contour detector 400. 
If an only if the lane detector 500 determines that the 
road contour indicated by the road contour signal Sre is 
correct does the lane detector 500 pass the road contour 40 
signal Sre to the local position detector 600. 

The lane contour detector 400 updates the road 
contour signal Sre based on the feedback of the road 
judgement signal Sj, and again outputs the road contour 
signal Sre to the lane detector 500. The lane contour 4S 
detector 400 continues to update the road contour signal 
Sre according to the road judgement signal Sj until the 
lane detector 500 determines that the road contour sig- 
nal Sre represents a correct road contour, and the road 
contour signal Sre is thus output only after it is deter- so 
mined to represent a correct road contour. 

The local position detector 600 thus receives the 
road contour signal Sro from the lane detector 500, and 
a vehicle condition signal Sc input from the ECU 700. 
Based on the road contour signal Sre. which thus rep- ss 
resents a reliable road contour, and the vehicle condition 
signal Sc indicative of the operating status of the motor 
vehicle, the local position detector 600 delects the mov- 



ing or stationary position of the automobile AM, and thus 
generates a position detection signal Sp. It should be 
noted that a method and apparatus for generating an 
ECU control signal for controlling vehicle operation, and 
navigation and control signals for a navigation appara- 
tus, based on this position detection signal Sp are dis- 
closed in Japanese patent application H8-1 02892 
(1996-102892; filed 24 April 1996). and United States 
Patent Application 08/637,417 (file 25 April 1996), both 
entitled and assigned to the same assignee as the 
present application. 

A simplified side view of an automobile AM in which 
the digital imaging apparatus 100 of the present inven- 
tion is installed is shown in Fig. 4. The optical axis Ax of 
the digital imaging apparatus 100 is aligned with the di- 
rection of travel Z of the automobile AM on the road, but 
is oriented downward from the horizontal by a known 
number of degrees q. The view angle of the digital im- 
aging apparatus 100 is also wide enough to capture the 
road area immediately betow and In front of the automo- 
bile AM in the perspective image Vi. 

An exemplary perspective image Vi of the fonward 
direction as seen from the automobile AM and repre- 
sented by the digital image signal Si captured by the dig- 
ital imaging apparatus 100 is shown in Fig. 5. It is as- 
sumed in this image that the automobile AM is travelling 
in the left lane of a two lane road. The two lanes are 
defined by the three lane markers Lml , Lm2, and Lm3. 
More specifically, the right lane Is delineated by lane 
markers Lm3 and Lm2. and the left lane by lane markers 
Lm2 and Lml. The lane markers are commonly applied 
using a high visibility paint such as white or yellow is 
commonly used. In the image shown in Fig. 5 the area 
at the bottom of the perspective image Vi is an image of 
the road surface directly below the front of the automo- 
bile AM. 

It should also be noted that the perspective Image 
Vi captured by the digital imaging apparatus 100 is a 
typical video image containing pixels of varying lumi- 
nance, density, and hue, and that line drawing represen- 
tations appearing as simple contour lines of objects in 
the viewing area are used in Fig. 5 because of simple 
drawing limitations. 

The preferred embodiments of the present inven- 
tion are described next below with reference to the ac- 
companying figures. 

First Embodiment 

Fig. 2 is a detailed block diagram of the spatial fre- 
quency separator 200. lane area detector 300, lane con- 
tour detector 400. lane detector 500, and local position 
detector 600 of the digital imaging apparatus 1 00 shown 
in Fig. 1. 

The spatial frequency separator 200 comprises a 
low frequency extractor 202 for extracting the low fre- 
quency component of the digital image signal SI and 
generating a tow spatial frequency signal SL. and a high 
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* frequency extractor 204 for extracting the high frequen- 
cy compenent and generating a high spatial frequency 
signal SH. 

The low frequency extractor 202 uses the lumi- 
nance data of the digital image signal Si to generate an 5 
image VL of the low spatial frequency component only, 
and in an exemplary embodiment of the invention a two- 
dimensional low pass filter. 

The high frequency extractor 204 similarly uses the 
luminance data of the digital image signal Si to generate 
an image VH of the high spatial frequency component 
only, and in an exemplary embodiment of the invention 
a two-dimensional high pass filter 

More specifically, the low frequency extractor 202 
and high frequency extractor 204 separate the perspec- '5 
tive image Vi captured by the digital imaging apparatus 
1 00 into a low frequency image VL and a high frequency 
image VH such that there is no overlap in the frequency 
components of the images. 

The road area detector 300 comprises a lane bright- 20 
ness setter 302, a lane image extractor 304, and a lane 
region setter 306. The lane area detector 300 is up- 
stream of the lane detector 500, which relies on a region 
determined by the area detector 300 and a contour ex- 
tracted by contour detector 400. In this sense, it maybe 2S 
more appropriate for the area detector 300 to be a road 
surface area detector and each of the internal compo- 
nents thereof to be road surface devices. It makes more 
sense for the lane contour detector 400 to be a 'road 
contour detector" for contrast with the lane contour ex- 30 
tractor 606. Hereinafter, uniformly used are the following 
terms: road area detector 300. road brightness setter 
302, road image extractor 304. road region setter 306. 

The road brightness setter 302 is connected to the 
low frequency extractor 202 of the spatial frequency 3$ 
separator 200, and to the ECU 700. and receives re- 
spectively therefrom the low spatial frequency signal SL 
and the vehicle condition signal Sc. Based on the veloc- 
ity signal Sv component of the vehicle condition signal 
Sc. the road brightness setter 302 sets road brightness 40 
levels Bmin and Bmax, which define the luminance lev- 
els of the pixels in the area of the road in the tow spatial 
frequency signal SL, and outputs a road brightness sig- 
nal SBr indicative of the threshold luminance together 
with the low spatial frequency signal SL. 45 

Based on the low spatial frequency signal SL and 
road brightness signal SBr (Bmax) supplied from the 
road brightness seller 302, the road image extractor 304 
extracts the road region of the low spatial frequency sig- 
nal SL, and generates a road extraction signal Srs. The 50 
road image extractor 304 thus supplies the low spatial 
frequency signal SL, road brightness signal SBr, and 
road extraction signal Srs to the road region setter 306. 

The road region setter 306 then determines what 
part of the low spatial frequency signal SL represents 55 
the region in which the road is located based on the road 
extraction signal Srs, and thereby generates the road 
region signal Srr supplied to the lane detector 500. 



The lane contour detector 400 comprises a thresh- 
old setter 402. contour extractor 404. and contour ex- 
traction region limiter 406. Connected to the high fre- 
quency extractor 204 of the spatial frequency separator 
200. the threshold setter 402 sets the threshold value 
Eth extracted by edge detection processing the contour 
points of the image representing the road in the high 
spatial frequency signal SH, and outputs a contour 
threshold signal Sth indicative of these threshold values 
Eth with the high spatial frequency signal SH. 

Based on the contour threshold signal Sth, the con- 
tour extractor 404 extracts the contour of the road image 
in the high spatial frequency signal SH, and thus gener- 
ates the road contour signal Sre supplied to the lane de- 
tector 500 and to the contour extraction region limiter 
406. 

Based on the road contour signal Sre. the contour 
extraction region limiter 406 produces a region limiting 
signal Sr restricting the region in the high spatial fre- 
quency signal SH to be used for road contour extraction, 
and feeds this region limiting signal Sr back to the high 
frequency extractor 204 of the spatial frequency sepa- 
rator 200. 

The high frequency extractor 204 thus extracts the 
high frequency component and generates the high spa- 
tial frequency signal SH from an area in the digital image 
signal Si restricted by the region limiting signal Sr. 

The contour extractor 404 further receives a road 
judgement signal Sj as feedback from the lane detector 
500. If the road contour signal Sre is determined by the 
lane detector 500 to not represent the true contours of 
the lane or road, the road contour signal Sre is updated 
to use the contour edge adjacent to the contour line cur- 
rently extracted as the tane contourasthe updated con- 
tour line, and this updated road contour signal Sre is 
then output to the lane detector 500. 

The lane detector 500 thus determines whether the 
road contour signal Sre represents the correct road con- 
tour edge based on the road region signal Srr, and gen- 
erates a road judgement signal Sj. In the exemplary em- 
bodiment of a lane detector 500 described herein, the 
road judgement signal Sj is preferably high when the 
correct road contour edge is extracted, and is otherwise 
low. In other words, when the road judgement signal Sj 
is low, the lane detector 500 stops outputting the road 
contour signal Sre, and causes the contour extractor 
404 to update the road contour signal Sre. As a result, 
the lane detector 500 outputs the road contour signal 
Sre to the local position detector 600 only when the road 
judgement signal Sj is high. 

More specifically, therefore, the contour extractor 
404 detects a series of projected contour points repre- 
senting candidates for the actual lane markers on the 
road using the high spatial frequency signal SH from the 
high frequency extractor 204. This series of projected 
contour points is output to the lane detector 500 as the 
road contour signal Sre. Using the road region data (i. 
e.. the road region signal Srr) detected by the road im- 
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age extractor 304, and the projected contour points (i. 
e., the road contour signal Sre) detected by the contour 
extractor 404. the lane detector 500 then detects the 
contour points of the lane markers. 

The local position detector 600 comprises a coordi- 
nate converter 602, matching detector 604. lane contour 
extractor 606. and in -lane position detector 608 

The coordinate converter 602 converts the front 
perspective image Vi to a birds'-eye image Vcc by ap- 
plying a coordinate conversion process to the road con- 
tour signal Sre. and thus produces a brrds*-cyG contour 
signal Sec. 

Based on the birds'-eye contour signal Sec supplied 
from the coordinate converter 602, the matching detec- 
tor 604 then detects whether the road surface for which 
a contour was detected is straight or curved to output a 
matching signal Sm indicative ol the matched road 
shape. 

Based en the matching signal Sm input from the 
matching detector 604, the lane contour extractor 606 
extracts a contour of the lane currently occupieO by the 
automobile AM to generate a lane contour signal Sic. 

Next, based on the lane contour signal Sic input 
from the lane contour extractor 606 and the vehicle con- 
dition signal Sc supplied from the ECU 700 the n lanc 
position detector 603 detects the position of tho auto- 
mobile AM within the occupied lane lo gonorHio tho po- 
sition detection signal Sp output thorotrom lo the ECU 
700. 

Note that the local position detector 600 gonorates 
the position detection signal Sp using a match dcloction 
operation based, in part, on the vchtcloconddion signal 
Sc supplied from the ECU 700, In an exemplary embod- 
iment of the present invention as doscribod lurtncr be- 
low with reference to Figs. 20 to 22. tho nrwitch detection 
operation of the matching detector 604 uses tho steering 
signal SsT component of the vehicle condition signal Sc. 
The invention shall not be so limited, however, and other 
methods can be alternatively used 

The operation of the essential parts o* a local posi- 
tioning apparatus LP according to a prclcrrod embodi- 
ment of the present invention is dcscnbod noxi below 
with reference to the accompanying Figs 5 through 25. 
The operation of the digital imaging apparatus 100 has 
already been described above with rcloroncc lo Fig. 5. 
The next step is therefore to describe tho edge extrac- 
tion operation of the high frequency oxtracior 204 of the 
spatial frequency separator 200 with rolcionco lo Figs. 
5 and 6. 

As described above, the high frequency extractor 
204 extracts the high frequency component of the digital 
image signal Si by applying a Sobel filter or other filtering 
process to the digital image signal Si. The filtering op- 
eration detects the edge pixels where there is a sudden 
change in pixel density. The extractor 204 then gener- 
ates a high spatial frequency signal SH representing an 
edge image Vh. To reduce the amount of data that must 
be filtered for this extraction, the filtering operation istim- 
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ited to a specifically limited part of the perspective image 
Vi. It is therefore possible to more quickly detect the per- 
spective image edge Vh data, and generate the high 
spatial frequency signal SH. 

More specifically, the perspective image Vi (Fig. 5) 
is divided into two parts, a top region St and a bottom 
region Sb, using a single horizontal line Ls. Note that 
the horizontal line Ls is aligned to the A-th vertical pixel 
counted from the bottom of the image. The area of the 
top region St is therefore Ph x (PV - A) pixels, and the 
area of the bottom region Sb is Ph x A pixels. The vertical 
position of this A-th pixel is preferably set to match the 
horizontal position of the vanishing point of perspective 
image Vi when the perspective image Vi is captured with 
the automobile AM en a level road. In the exemplary em- 
txxJiment of the invention, pixel A is set to the vertical 
position of a pixel representing an image positioned ap- 
proximately fifty meters away from the automobile AM 
when the perspective image Vi is captured directly by 
the digital imaging apparatus 100. 

Immediately after the local positioning operation of 
the present invention is started, the high frequency ex- 
tractor 204 applies the above high spatial frequency 
component extraction (filtering) operation to the bottom 
region Sb only to extract the edge pixels only from and 
generate the high spatial frequency signal SH only for 
the bottom region Sb. This obviously means that the 
high spatial frequency signal SH only contains edge pix- 
els extracted from the bottom region Sb of Ph x A pixels. 

The extracted edge image Vh represented by the 
high spatial frequency signal SH is shown in Fig. 6. The 
edge pixels of and near primarily the lane markers Lml , 
Lm2. and Lm3 are extracted from the bottom region Sb 
of the perspective image Vi (Fig. 5) as the edge pixels. 
Though not shewn in Fig. 6. it should be obvious that 
edge pixels from objects unrelated to the lane markers 
are also extracted. 

The threshold setter 402 thus receives the high spa- 
tial frequency signal SH comprising the extracted edge 
pixels from the low frequency extractor 202, and calcu- 
lates the threshold value Eth for effectively extracting 
the edge pixels from the contours of the lane markers 
Lm in the high spatial frequency signal SH using the fol- 
lowing equation (1). 



Eth = C * Emax + (1 - C) * Emean 



0). 



where Emax and Emean are the maximum density and 
the mean density values of the pixels on a particular hor- 
izontal line in the bottom region Sb of the perspective 
image Vi; and C is a constant where 0 < C < 1 . 

Tho contour extraction method of the contour ex- 
tractor 404 is described next referring next to Figs. 7 and 
8. An edge image Vh (SH) from which the contour ex- 
tractor 404 detects contour lines is shown in Fig. 7. 

The contour extractor 404 scans the pixels in the 
bottom region Sb of the high spatial frequency signal SH 
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using the threshold value Eth to extract the contour lines 
of the displayed lanes. This bottom region Sb is further 
divided into left and right bottom regions SbL and SbR 
by a vertical center line Lc. Note that the vertical center 
line Lc rises vertically from pixel Ph/2 on the bottom line s 
of the image in this exemplary embodiment, but the in- 
vention shall not be so limited. 

The contour extractor 404 compares the density of 
each pixel in the scanning area with the threshold value 
Eth, and extracts those pixels with a pixel density great- io 
er than or equal to the threshold value Eth as a contour 
pixel. The scanning order for this operation follows hor- 
izontal line Ls and works to the left and right of the ver- 
tical center line Lc along left and right horizontal lines 
LhL and LhR. After scanning along horizontal line Ls, ?5 
the contour extractor 404 drops down a particular 
number of pixels along the vertical center line Lc, and 
again scans to right and left to obtain the contour pixels 
on that horizontal line. 

In exemplary practice, however, the contour extrac- 20 
tor 404 scans first to the left or the right of the center line 
in the bottom region Sb to obtain the contour pixels on 
that side. For example, if the left side of the bottom re- 
gion Sb is scanned first, the contour extractor 404 scans 
top to bottom starting from horizontal line Ls at pixel A, 2S 
scanning lines at an interval of PLh pixels downward to 
the bottom of the image, and scanning every pixel right 
to left from the vertical center line Lc along the left hor- 
izontal line LhL. After obtaining all contour pixels on the 
bottom left sector of the image in this sequence, the con- 30 
tour extractor 404 then returns to the top horizontal line 
Ls to similarly scan the bottom right sector of the image 
in left to right sequence from the vertical center line Lc 
and obtain the contour pixels therein. 

Skipping the horizontal scanning position at a par- 3S 
ticular interval PLh results in extraction from an integer 
K number of horizontal lines Lh equivalent to the abso- 
lute value of Pv/PLh. This reduces the number of calcu- 
lations to be performed, and thereby increases the 
speed of contour pixel extraction. Note that the value of 
PLh is a natural number set in a pixel unit increment. 

When the first pixel with a particular density value 
is found on left or right horizontal lines LhL and LhR at 
a given vertical position, that pixel Pe is defined as a 
contour pixel, and contour pixel scanning for that hori- 45 
zontal line is skipped. Scanning then advances directly 
to the next horizontal line PLh pixels down. 

In other words, contour pixels are delected at the 
inside edges of the lane markers Lm1 and Lm2 on the 
left and right sides of the shown lane at the first horizon- 50 
tal line Lh counted down from the horizontal line Ls, as 
shown in Fig. 7. At the third and fourth lines, however, 
there is a gap between successive lano markers Lm2 
dividing the right and left lanes, and contour pixel detec- 
tion to the right side of the vertical center line Lc at these 55 
lines, or more specifically, at any vertical position be- 
tween these lane markers Lm2. will fail to detect a con- 
tour pixel from any lane marker Lm2 and may detect a 



contour pixel from the lane marker Lm3 at the edge of 
the adjacent lane. The edges of nnarkings other than the 
lane markers but causing a similar sudden change in 
pixel density, e.g., soil or even imaging flaws, will also 
be detected as contour pixels. Such contour pixels are, 
however, irrelevant to detecting the vehicle lanes, and 
are therefore simply a noise component insofar as the 
position detection operation of the present invention is 
concerned. As a result, these noise components are re- 
moved by the lane.detector 500 as described in further 
detail below. 

An extracted contour image Vre represented by the 
road contour signal Sre obtained as described above 
from the image in Fig. 7 is shown in Fig. 8. The detected 
contour pixels of the lane markers Lml , Lm2, and LmS 
are shown as contour signals ScL, ScR, and ScR' where 
ScR' is noise data. Contour pixel signal ScL' represents 
noise data extracted due to some other flaw, shadow, 
or marking apparent in the image. 

The method of limiting the detection area by means 
of the contour extraction region limiter 406 is described 
next below with reference to Figs. 9 and 10. As shown 
in Fig. 9, the contour extraction region limiter 406 de- 
fines a limited region RsL and RsR of a known width Wr 
to the right and left sides of the extracted contour signals 
ScL and ScR based on the road contour signal Sre. The 
region limiting signal Sr identifies these limited regions 
RsL and RsR, and is output to the high frequency ex- 
tractor 204 of the spatial frequency separator 200. How- 
ever, when the contour extractor 404 is unable to extract 
the lane contour and an error signal See (not shown in 
the figures) in place of the road contour signal Sre, the 
extractbn region is not limited. 

The width Wr of the limited region is determined with 
respect to the amount of movement in the x-axis (hori- 
zontal) direction of the lane markers in the image as a 
result of lateral movement of the automobile AM. More 
specifically, Wr is the distance that lane marker Lml can 
move in the perspective image Vi in one system cycle, 
i.e., 33 mS in an exemplary embodiment of the inven- 
tion. It therefore follows that the limit width Wr increases 
towards the bottom of the image. 

As shown in Fig. 10, the high frequency extractor 
204 limits the area for edge extraction in the bottom re- 
gion Sb of the perspective image Vi to the limited regions 
RsR and RsL based on the region limiting signal Sr. in 
addition to reducing the amount of data to be processed 
and therefore increasing the processing speed, this 
process also suppresses noise components from out- 
side the lane contours, and therefore improves the lane 
tracking ability. 

When a noise component is sufficiently removed 
from the limited regions RsR and RsL, as is the contour 
line Scr' of the opposing lane marker Lm3 described 
above, the noise component is removed by the contour 
extraction region limiter 406. However, when a contour 
noise component is near a lane contour, as is the con- 
tour line ScL' in Fig. 8. the noise component will be within 
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the limited region Rsr In this case the noise component 
cannot be removed by the contour extraction region lim- 
iter 406. As a result, these noise components must be 
removed by the lane detector 500, as will be described 
later below. s 

The method of generating a low spatial frequency 
Image by means of the low frequency extractor 202 of 
the spatial frequency separator 200 is described next 
below with reference to Fig. 11 . 

The low frequency extractor 202 generates the low io 
spatial frequency signal SL by low-pass filtering the dig- 
ital image signal Si. The subject of the perspective im- 
age Vi appears as a fuzzy grouping of pixels of various 
luminance levels in the low frequency image VL repre- 
sented by the low spatial frequency signal SL The low is 
frequency image VL represented by the low spatial fre- 
quency signal SL is thus a luminance image. To reduce 
the processing load of the low spatial frequency signal 
SL generating operation, the luminance pixels are there- 
fore extracted from a limited part of the perspective Im- 20 
age Vi, specifically, the Ph x A pixel bottom region Sb. 
This is similar to the method of generating the high spa- 
tial frequency signal SH described above. 

The luminance of the pixels in the road region of the 
low frequency image VL tends to increase with the dis- 2S 
tance from the automobile AM, i.e., the distance from 
the digital imaging apparatus 100. The same is true of 
the luminance level of pixels in image areas correspond- 
ing to the road shoulder, drainage channels, and guard 
rails. In the image in Fig. 11, the pixels at vertical pixel 30 
level A, i.e., along horizontal line Ls, have the highest 
luminance level of any pixels associated with the same 
object, and the pixels toward the bottom of the image 
have the lowest luminance level. 

The luminance level of pixels at the same vertical 55 
height but associated with different objects will, howev- 
er, vary from object to object. More specifically, the lu- 
minance will vary according to the reflectance of the 
subject material, and the angle of incidence of light from 
the light source. The above described relationship be- "^o 
tween luminance and distance, i.e.^ that luminance in- 
creases with distance, is, however, retained. 

The low frequency image VL is thus produced only 
from the bottom region Sb of the perspective image Vi 
obtained by the digital imaging apparatus 100 imaging 
primarily the area directly in front of the vehicle. Further- 
more, a majority of the pixels in the low frequency image 
VL represent the road surface, and this tendency is 
therefore substantially applicable to all pixels in the tow 
frequency image VL. The pixels for the road surface de- so 
tected along the horizontal line Ls at the farthest point 
from the digital imaging apparatus 100 in the low fre- 
quency imago VL are therefore defined as the farthest 
road pixels Pf, and the luminance level of those pixels 
is defined as maximum luminance Bmax. The pixels for ss 
the road surface detected at the bottom of the image 
nearest the digital imaging apparatus 100 in the low fre- 
quency image VL are therefore defined as the nearest 
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road pixels Pn. and the luminance level of those pixels 
as minimum luminance Bmin. 

Note that the location of the farthest road pixels R 
and nearest road pixels Pn can be freely set as required. 

A histogram of all pixels in the low frequency image 
VL is shown in Fig. 12 with luminance levels plotted on 
the horizontal axis and the number of pixels with a given 
luminance level plotted on the vertical axis. As will be 
known from Fig. 12. a majority of the pixels in the low 
frequency image VL are road surface pixels with lumi- 
nance between minimum luminance Bmin and maxi- 
mum luminance Bmax. The luminance range between 
minimum luminance Bmin and maximum luminance 
Bmax is therefore called the road luminance range Br, 
and the pixels in this road luminance range Br are called 
road pixels Pr. 

When the digital imaging apparatus 100 is installed 
at substantially the front center of the automobile AM as 
described with reference to Fig. 4, the positions of the 
farthest road pixels Pf and nearest road pixels Pn are 
set at appropriate vertical positions Vpf and Vpn in the 
image on the vertical center line Lc. The road surface 
occupying the area between Vpf and Vpn is then indi- 
cated by the road pixels Pr. The position of Vpf can 
therefore be thought of as the farthest distance at which 
the road surface is detected, and the position of Vpn as 
the nearest distance at which the road surface is detect- 
ed. 

The position of the farthest road pixels Pf is set ap- 
propriately on the farthest road detection distance Vpf 
according to the position at which the digital imaging ap- 
paratus 100 is mounted on the automobile AM. and the 
position of the nearest road pixels Pn is likewise set ap- 
propriately on the nearest road detection distance Vpn. 

The farthest road detection distance Vpf and near- 
est road detection distance Vpn can be set according to 
the vehicle speed and the processing capacity (speed) 
of the local positioning apparatus LP as described fur- 
ther below. 

The relationship between the luminance of only 
road pixels in the low frequency image VL and the dis- 
tance of the road surface from which the pixels are cap- 
tured is shown in Fig. 13. The values shown in Fig. 13 
are for pixels in an image experimentally obtained while 
driving on a straight section of highway on a sunny day 
with the digital imaging apparatus 100 mounted on the 
front of the automobile AM. The horizontal axis indicates 
the horizontal scanning position (Pv) in the low frequen- 
cy image VL, i.e., the relative distance D of each pixel 
from the digital imaging apparatus 100, and the vertical 
axis indicates the luminance B at that distance. That the 
distance - luminance relationship is not a flat line but 
has a right ascending slope is a result of lane markings, 
soiling, and color and luminance spots caused by re- 
flectance from obstructions on the road. As already de- 
scribed above, the area represented by a single pixel 
increases as the horizontal scanning position is elevat- 
ed, meaning that the scanning position becomes more 
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distant from the image pickup 100. However, the rela- 
tionship between imaging distance and pixel luminance 
can be expressed by equation (2) below. 



B= A'D+C 



(2) 



Note that values of A = 0.41 28 and D = 108.56 have 
been experimentally obtained on straight lanes of an ex- 
pressway. 10 

By correctly setting C and C" so that C* > C > C, 
and obtaining parallel lines of: 



and 



B = A-D + C 



B = A * D + C 



(3). 



(4). 
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above and below the line represented by cqjation (2). 
the range of luminance values of pixels at a particular 
scanning position can be obtained. 

The method whereby the road bnghtnoss 6ottor302 2S 
determines the luminance level of tho road surface is 
described next below with reference to F^g 14 Based 
on the relationship between the imagmg distance and 
pixel luminance described with referonco lo Figs 11 to 
1 3, the farthest road pixels Pf and nearest road pixels 30 
Pn in the low spatial frequency signal SL arc sot to the 
pixel positions corresponding to tho distance defining 
the road area to be used for positioning doioction. The 
pixel luminance at the farthest road pixels Pf is then de- 
fined as maximum luminance Bmax. and the pixel lumi- 3S 
nance at the nearest road pixels Pn is dofinod as the 
minimum luminance Bmin to generate the road bright- 
ness signal SBr Note that the road brightness signal 
SBr may be set to have luminance levels m the range 
defined by equations (3) and (4) atx)vo Tho locations 40 
of the farthest road pixels Pf and the nearest road pixels 
Pn in the low frequency image VL arc expressed as Pf 
(xPf ,yPf) and Pn(xPn.yPn) using the coordiriato system 
shown in Fig. 3. 

By setting the position of the farthest road pixels Pf 45 
from which the maximum luminance Bmax is detected 
to an appropriate position according to Iho processing 
speed Tp of the local positioning appaiatus LP and the 
speed of the automobile AM, high precision lane detec- 
tion is possible based on the operating conditions of the . so 
automobile AM on which the local posiiionng apparatus 
LP is installed. 

For example, by moving the farthest road pixels R 
vertically upward in the low frequency innago VL accord- 
ing to the velocity signal Sv of the autorrxjbilo AM, the ss 
detection precision at points far from the aulonr>obile AM 
can be maintained even when the autonr>obile AM is 
travelling at a high rate of speed. When the automobile 



AM is travelling slowly, lane detection precision can be 
improved by moving the position of the farthest road pix- 
els Pf closer. Because the vehicle will pass the detection 
point while the detection process is in progress when 
the vehicle is travelling fast and the farthest road pixels 
R is close, it is preferable to detect the maximum lumi- 
nance Bmax at a suitable distance from the vehicle 
based on the velocity signal Sv and the processing ca- 
pacity of the local positioning apparatus LP 

When the automobile AM is travelling on a straight 
road, the farthest road pixels R can be set on the vertical 
center line Lc of the low frequency Image VL. However, 
when the automobile AM is on a curving road, setting 
the farthest road pixels Pf on the vertical center line Lc 
may result in the maximum luminance Bmax being ex- 
tracted from a guard rail or other non-road subject within 
the low frequency image VL. 

In such cases, it is necessary to estimate the cur- 
vature of the road from the steering angle of the auto- 
mobile AM so that only pixels within the image area of 
the road surface in the low frequency image VL are used 
for extracting the maximum luminance Bmax. This can 
be accomplished by moving the farthest road pixels Pf 
horizontally within the low frequency image VL accord- 
ing to the steering signal Ss so that the farthest road 
pixels Pf are within the road surface area. 

By moving the position of the farthest road pixels R 
freely horizontally and vertically within the low frequency 
image VL based on the velocity signal Sv and steering 
signal Ss. the farthest road pixels R can be adjusted to 
the road image area. Note that a value other than the 
velocity signal Sv can be alternatively used, including a 
distance value (vertical image position value) entered 
by the user. Data on the direction of travel (straight or 
curved) supplied by a navigation system based, for ex- 
ample, on map data, can also be used in place of the 
steering signal Ss. More specifically, the pixels of the 
subject area can be reliably captured by moving or fixing 
the positions of the farthest road pixels R and nearest 
road pixels Pn in the low frequency image VL as neces- 
sary. 

The method whereby the road image extractor 304 
extracts an image of the road surface is described next 
below with reference to Figs. 15 to 17. The road image 
extractor 304 scans the pixels in the bottom region Sb 
of the low spatial frequency signal SL based on the pixel 
luminance relationship to the scanning position shown 
in Fig. 1 3 and defined by equations 2, 3, and 4 lo extract 
the pixels in the road image area. The road image con- 
tours extracted by the road image extractor 304 are sim- 
ilar to the contour lines of the lanes obtained by the con- 
tour extractor 404 and described with reference to Fig. 
12. The road image extractor 304, however, extracts the 
pixels with a luminance level between maximum lumi- 
nance Bmax and minimum luminance Bmin, i.e., pixels 
satisfying equations 2. 3, and 4 at each horizontal scan- 
ning position Pv rather than using an edge extraction 
method as does the contour extractor 404. 
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Region pixel sets Elm. Erm, and Erm' correspond- 
ing to the contour pixels ScL, ScR, and ScR' in Fig. 8 
are obtained at the inside circunnference part of lane 
markers Lm1. Lm2. and Lm3. It should be noted that 
lines representing the inside edges of the lane markers s 
Lm1, Lm2, and Lm3 can be obtained as necessary 
based on the region pixel sets Elm, Erm, and Erm' using 
a least squares method or other suitable method such 
as the Hough conversion described further below with 
reference to Figs. 20 and 22. An example of Hough con- io 
version of pixel set Elm is shown in Fig. 16, and an ex- 
ample of a linear approximation of the road surface is 
shown in Fig. 17. Note that linear approximation effec- 
tively removes the inside edge part set Erm' of the lane 
marker Lm3 of the adjacent lane. is 

The method whereby the road region setter 306 de- 
fines the road surface area is described next below with 
reference to Fig. 18. 

The road region setter 306 sets road edge areas 
AeL and AeR based on the road extraction signal Srs 2o 
by adding a known horizontal width Wa to both right and 
left sides of the extracted inside edge areas Elm and 
Erm. The road region signal Srr output to the lane de- 
tector 500 is indicative of these road edge areas AeL 
and AeR. When the road region setter 306 cannot ex- 25 
tract the road area and the error signal See is output In 
place of the road extraction signal Srs, the road edge 
areas AeL and AeR are not set. Note that while the op- 
eration of the road image extractor 304 is similar to that 
of the contour extraction region limiter 406, the limit 30 
width Wa can be an appropriately defined value based 
on the values C. C\ and C" shown in equations 2 to 4 
above. 

Using the low spatial frequency signal SL from the 
low frequency extractor 202, the road image extractor 35 
304 divides the low frequency image, or luminance im- 
age, VL into areas of pixels having a particular relation- 
ship based on the luminance distribution. Certain road 
knowledge is then referenced and applied to these di- 
vided regions to detect the road area. 40 

The "road knowledge" used herein includes know- 
ing that the area containing the farthest road pixels Pi 
and nearest road pixels Pn in the perspective image Vi 
captured by the digital imaging apparatus 100 Is within 
the road area. Other knowledge includes knowing that ^5 
the vanishing point at which the right and left lane mark- 
ers intersect does not change suddenly, and that the pix- 
el area of the road surface is substantially constant. The 
area containing the nearest road pixels Pn and the ad- 
jacent area can therefore be defined as the road area. so 

The operation ol the lane detector 500 is described 
next. 

The lane detector 500 determines whether the con- 
tour lines ScR and ScL described by the road contour 
signal Sre from the lane contour detector 400 (Fig. 8) ss 
are within the road edge areas AeR and AeL described 
by the road region signal Srr. To accomplish this the lane 
detector 500 determines whether the ScR value is within 
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AeRiWa at the same horizontal scanning line position 
(distance). If ScR is within this range, the road contour 
signal Sre is determined to represent a correct road con- 
tour. The road judgement signal Sj is therefore output 
low. and the road contour signal Sre is passed through 
to the local position detector 600. 

However, if ScR is outside the range AeI=l±Wa. the 
road contour signal Sre is determined not to represent 
a correct road contour, but to be within a noise contour 
component The road judgement signal Sj is therefore 
output high, and the road contour signal Sre is not out- 
put. 

It is therefore possible as described above to pro- 
duce a low frequency image and a high frequency image 
with no spatial frequency overlap between the images 
from the same luminance image data, and separately 
detect from these images the road area and lane marker 
contour points. As a result, lane markers can be detect- 
ed with high precision. 

The coordinate conversion principle used by the co- 
ordinate convertor 602 to convert the extracted contour 
image Vre is described briefly below with reference to 
Figs. 3 and 4. 

As described above, the coordinate convertor 602 
converts the coordinate system of the road contour sig- 
nal Sre to convert the extracted contour image Vre rep- 
resented by the road contour signal Sre to a birds'-eye 
image Vcc. However, the perspective representation of 
objects in the extracted contour image Vre results in 
shape distortion that increases with the distance from 
the digital imaging apparatus 100. This means that, for 
example, when there are two identical objects with one 
placed at a greater distance from the digital imaging ap- 
paratus 100, the farther object will appear smaller than 
the nearer object. This shape distortion increases with 
the distance from the digital imaging apparatus 100 or 
automobile Af^. 

The coordinate system of perspective images Vi, 
Vh, and Vre is shown in Fig. 3. The birds'-eye view co- 
ordinate system referenced to the automobile AM is 
shown in Fig. 4, The x-axis of the image coordinate sys- 
tem (x,y) is parallel to and oriented in the same direction 
as the X-axis in the coordinate system of the birds'-eye 
image Vcc, and the y-axis is inclined to the Y-axis q de- 
grees because the optical axis Ax of the digital imaging 
apparatus 100 is inclined q degrees from level. The X- 
axis and Z-axis therefore define a level plane Pd that Is 
parallel to the travelling surface, i.e., road surface, of the 
automobile AM. The Y-axis and y-axis are aligned: and 
the X-axis is perpendicular to the surface plane of Fig. 
4. The origin O matches the original of the coordinate 
system of the extracted contour image Vre. The optical 
axis Ax of the digital imaging apparatus 100 passes 
through the origin O. 

The coordinates of the extracted contour image Vre 
can be converted to the coordinates of the birds'-eye 
image Vcc using the following equations. 
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X = (x/F) (Z cos e - Y sin 9) 
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(5). 



22 



X = FX/(2 cos 9 + H sin 9) 



(8) 



Z = Y(F cos e + Y sin 9) 



(6) 



z = F(H cos 6 + Z sin 9)/(Z cos 9 - H sin 9) (9) 



and 



Y = -H 



(7). 



where F is the focal distance of the optical lens of the 
digital imaging apparatus 100; 9 is the angle between 
the Z-axts (horizontal axis) and the optical axis Ax of the 
lens; and H is the distance from the road surface to the 
origin of the optical axis of optical lens. The values of 9 
and H are preferably set to obtain a perspective image 
Vi of the area to the front of the automobile AM as de- 
scribed wilh reference to Fig. 5. In an exemplary em- 
bodiment of the present invention, 9 is 6 degrees, and 
H is 1 .2 meters. 

The birds'-eye contour signal Sec is then obtained 
by converting the coordinates of the road contour signal 
Sre by converting the distances in the extracted contour 
image Vre using the above equations 5 to 7. Horizontal 
distances in this birds'-eye contour signal Sec are accu- 
rately represented irrespective of the Z-axis direction 
from the digital imaging apparatus 100. f^ore precisely, 
the coordinate-converted image Vcc is not a birds'-eye 
image, but is a plan view of the road surface captured 
from a plane parallel to the road surface. Furthermore, 
image Vcc always represents the road surface as a flat 
surface even when, for example, there are low rises or 
obstructions in the road. This error/difference (low rises 
or obstructions), however, does not degrade the detec- 
tion precision of the local positioning apparatus accord- 
ing to the present invention. This is because the posi- 
tioning detection operation of the present invention uses 
a relatively near distance of only approximately fifty me- 
ters forward as the farthest forward point used for de- 
tection and control, and bumps and obstructions on the 
road surface can therefore be ignored. 

If a full birds'-eye image of the road surface, i.e.. a 
plan view image of the road surface captured perpen- 
dicularly to a horizontal plane perpendicular to the ver- 
tical line with respect to the gravitational axis is required, 
it can be obtained by various methods. One such meth- 
od is to not fix the optical axis Ax of the digital imaging 
apparatus 100 to the automobile AM, and use a gyro- 
scope or other automatic attitude control device to al- 
ways obtain the image Vi from a constant attitude to the 
horizontal. 

For reference, reverse coordinate conversion from 
a birds'-eye view coordinate system to a perspective 
view coordinate system can be achieved using equa- 
tions 8 and 9 below. 
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The contour image Vcc represented by the coordi- 
nate-converted birds'-eye contour signal Sec is shown 
in Fig. 19. The lane markers Lml. Lm2, and Lm3 are 
represented by corresponding edge contours ScL, ScR. 
and ScR'. As will be known by comparison with the ex- 
tracted contour image Vre in Fig. 8, the edge contours 
ScL, ScR, and ScR' are parallel throughout Fig. 1 9, and 
represent the actual lanes on the road. In other words, 
the coordinate -converted birds'-eye contour signal Sec 
contains correct dimension information for the subject. 

Based on the coordinate-converted birds'-eye con- 
tou r signal See, the matching detector 604 connected to 
the coordinate convenor 602 obtains line segments or 
arcs matching the edge contours ScL, ScR, and ScR' 
by applying the following equations. 

The first step is a Hough conversion of the pixel data 
for the edge contours ScL, ScR, and ScR' of the lane 
markers in the birds'-eye contour signal See. This is ac- 
complished by applying equation (10) below separately 
to the corresponding contour lines. 



p = X coscl) + Z sin(() 



(10) 



where p is the distance between the origin O and the 
pixel in the Z-X coordinate system, and 4» is the angle 
between the X-axis and a line joining the origin O and a 
particular pixel. The following equation (11) is then ob- 
tained after calculating equation (10). 



X - (p - Z sin(t>)/ cos(J) 



(11) 



A group of curves is obtained for each contour line 
by scanning the birds'-eye contour signal Sec and con- 
verting the contour line data to parametric space based 
on the above Hough conversion. The method of deter- 
mining whether a contour line is a straight line segment 
or arc based on this group of curves is described below 
with reference to Figs. 10 to 22. 

Typical patterns obtained when the Hough-convert- 
ed line data is a straight line is shown in Fig. 20. Note 
that ideally each of the curves intersect at a single point 
Cp. As a practical matter, however, the lane markers Lm 
and contour lines are not usually completely straight 
lines, and convergence at a single point as shown in Fig. 
20 thus rarely occurs. This problem can be resolved by 
noting that the curves tend to intersect at the same point, 
and investigating the frequency Fc that each curve in- 
tersects at each point (pixel) in the parametric space. If 
this frequency Fc is greater than a particular threshold 
value Eth, that point is defined as the unique point of 
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intersection Cp. The corresponding contour line is then 
defined as a straight line, and the appropriate equation 
for a line is obtained. 

Typical patterns obtained when the Hough-convert- 
ed line data Is a curve is shown in Figs. 21 and 22. As 
shown in these figures, there is neither no point of inter- 
section (Fig. 21 ), or a group of curves intersect at a plu- 
rality of points Cpn (Cpl to Cp6) as shown in Fig. 22. 
Note that n in this case is an integer value. The frequen- 
cy of intersection Fc is again obtained. If there is no point 
where a new frequency threshold Fth' is exceeded, the 
group of curves is determine to not intersect, and the 
corresponding contour line Is determined to be a curve 
(arc). The appropriate equation for a curve is then ob- 
tained. 

It should be noted that these equations make it pos- 
sible to determine particular dimensional features of the 
road, i.e.. the lane that the automobile AM is either fol- 
lowing or stopped in. 

As shown in Fig. 2, the lane contour extractor 606 
is connected to the matching detector 604, and receives 
therefrom the matching signal Sm. Note that the match- 
ing signal Sm also contains contour line data ScR', 
which is obtained from noise components as described 
above. 

The lane contour extractor 606 compares the di- 
mensional features data contained in the matching sig- 
nal Sm with particular predetermined data such as the 
vehicle width, lane width, and the pattern of the center 
line lane marker Lm2 to remove contour data ScR' that 
Is not appropriate to the target lane. The remaining con- 
tour line data ScR and ScL corresponding to the current 
lane is output as the matching signal Sm. If there is no 
data corresponding to the lane in the matching signal 
Sm, an error signal See (not shown in the figures) is out- 
put. 

The filtering operation of the lane contour extractor 
606 is described next with reference to Fig. 19. 

This birds'-eye image Vcc contains three contour 
lines ScL. ScR, and ScR'. Counting down from the top 
of the image, the left two contour lines ScL and ScR are 
a contour line pair describing a single lane Lm down to 
the second pixels. The center line lane marker Lm2 in 
this image, however, is a divided line, and the contour 
line ScR' at the far side of the adjacent lane is therefore 
extracted from the birds'-eye contour signal Sec at pix- 
els 3 to 5. The distance between ScL and ScR' in this 
case is obviously greater than the width of a single lane, 
but it is still not known at this point whether contour line 
ScL or ScR' represents noise. 

From pixel 6 to 11 . however, contour lines ScL and 
ScR are obviously paired. The left contour line ScL is 
also determined by the matching detector 604 to match 
a single arc ScLm. and pixels 3 to 5 furthermore corre- 
sponding to lane marker Lml . The three pixels from pix- 
el 3 to 5 in contour line ScR' are therefore ignored as 
noise. The two contour lines ScL and ScR are therefore 
extracted as a correct contour line pair. The matching 
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contour lines ScLm and ScRm are thus selected, and 
output with the information of selected contour lines as 
the lane contour signal Sic. 

Combined with the road area evaluation result out- 
put as road contour signal Sre by the lane detector 500, 
the lane contour extractor 606 thus effectively elimi- 
nates noise components, such as other markings and 
soiling on the road, that are unrelated to lane definition 
from the lane contour signal Sic. 

The extracted lane contour image Vie represented 
by the road contour signal Sle after noise removal by 
the lane contour extractor 606 is shown in Fig. 23. 

The operation of the In-lane position detector 608 
is described next with reference to Fig. 23. 

Using the lane contour signal Sic, the In-lane posi- 
tion detector 608 obtains the relationship between a cir- 
cle or line tangential to or intersecting the right and left 
contour lines ScLm and ScRm to obtain the points of 
intersection HL and HR at Z = 0. The coordinates for the 
midpoint of this line segment, i.e., the coordinates (HO, 
O) of the line segment HL-HR, are then obtained. Note 
that the camera is located at (0.0). Therefore, if the dig- 
ital imaging apparatus 1 00 is mounted at the widthwise 
center of the automobile AM. the position of the auto- 
mobile AM relative to the lane can be expressed as the 
lateral displacement -HC from the lane center. 

The overall operation of a local positioning appara- 
tus LP according to an exemplary embodiment of the 
present invention is described next below with reference 
to the flowchart in Fig. 24. 

Operation starts after a digital image signal Si is 
gerierated by means of the digital imaging apparatus 
100 capturing a picture of the view to the front of the 
automobile AM (block #1). 

The spatial frequency separator 200 then extracts 
the low spatial frequency signal SL and high spatial fre- 
quency signal SH from the digital image signal Si (block 
#3). 

The threshold setter 402 then sets a threshold value 
Eth based on the edge pixel density In the high spatial 
frequency signal SH. and generates the contour thresh- 
old signal Sth (block #5). 

Based on the contour threshold signal Sth, the con- 
tour extractor 404 extracts the lane marker contours us- 
ing the threshold value Eth derived from the high spatial 
frequency signal SH, and outputs the road contour sig- 
nal Sre (block #7). 

Whether an error signal See is output in place of the 
road contour signal Sre is then determined (block #9). 
If the error signal See is output, contour lines have not 
been extracted. A YES is therefore returned and the pro- 
cedure loops back to block #1. If NO is returned, the 
procedure advances. 

Based on the road contour signal Sre. the contour 
extraction region limiter 406 then defines the area in the 
high spatial frequency signal SH to be used for contour 
extractbn. and outputs a region limiting signal Sr to the 
high frequency extractor 204 (block #11 ). This feedback 
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loop (see Fig. 2) causes the high frequency extractor 
204 to extract the high frequency connponent in the area 
defined by the region limiting signal Sr within the digital 
image signal Si at block #3 above, and generate the high 
spatial frequency signal SH from this limited region. 

The road brightness setter 302 then sets a road sur- 
face luminance area (Bmax, Bmin) in the low spatial fre- 
quency signal SL resulting from block #3, and outputs 
a road brightness signal SBr (block #13). 

Based on the road brightness signal SBr, the road 
image extractor 304 then extracts the road surface im- 
age from the low spatial frequency signal SL to generate 
the road extraction signal Srs (block #15). 

Next, based on the road brightness signal SBr and 
road extraction signal Srs. the road region setter 306 
generates a road region signal Srr describing the road 
edge areas AeR and AeL (block #17). 

Whether the error signal See is output In place of 
the road region signal Srr is then detected (block #19). 
If the road region has not been delecled, YES is re- 
turned and the procedure loops back to block #1 . If NO 
is returned, the procedure advances. 

The lane detector 500 then determines whether the 
extracted contour lines described by the road contour 
signal Sro arc valid based on the road region signal Srr 
(block #21 ). If the result is NO, the contour extractor 404 
is driven again by means of the road judgement signal 
Sj to update the road contour signal Sre. The sequence 
from block #7 to block #21 is thus repeated until valid 
contour lines are extracted. The road contour signal Sre 
is passed to the local position detector 600 only once 
the contour lines are determined to be valid (correct). 

The coordinate converter 602 then coordinate con- 
verts the extracted contour image Vre of the road con- 
tour signal Sre to a birds'-eye image Vcc to generate a 
birds'-eye contour signal Sec (block #23). 

The matching detector 604 then uses a Hough con- 
version or other appropriate conversion method to 
match an equation describing the internal contour lines 
of the birds'-eye contour signal Sec to straight line seg- 
ments or arcs (block #25). The matching detector 604 
also outputs a matching signal Sm representing the 
matching line segments or arcs. 

Whether an error signal See is output in place of the 
matching signal Sm is then determined (block #27). If 
there are no matching line or arc segments, YES is re- 
turned and the procedure loops back to block #1 . If NO 
is returned, the procedure advances. 

Dimensional features of the matching contour lines 
indicated by the matching signal Sm are then compared 
with the dimensional features of the automobile AM by 
the lane contour extractor 606 (block #29). The tane 
contour extractor 606 thus extracts a pair of lane contour 
lines descnbing the lane currently occupied by the ve- 
hicle, and generates a lane contour signal Sic. 

The current position of the automobile AM is then 
detected by the in-lane position detector 608 based on 
the lane contour signal Sic. and a position detection sig- 
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nal Sp is thus output (block #31). 

Note that, if the error signal See is detected in 
blocks #9, #19. or #27 above, the procedure is returned 
to the beginning (block #1 ) to achieve more precise lane 
detection and position detection by executing the pro- 
cedure for each Intervening block based on the high 
spatial frequency signal SH and low spatial frequency 
signal SL extracted at the same time. Depending on the 
desired precision, however, blocks #9. #19. or #27 can 
be omitted without substantially affecting the results 
achieved by the present invention. 

Second embodiment 

A local positioning apparatus according to a second 
exemplary embodiment of the present invention Is de- 
scribed next below with reference to Figs. 26 to 45. Fig. 
26 is a block diagram of a road area extraction appara- 
tus according to the present embodiment ot the inven- 
tion, which comprises a digital image pickup 11, bright- 
ness histogram generator 12, lane brightness separator 
13, and lane polygon detector 14. 

The digital image pickup 1 1 is typically a video cam- 
era used for capturing an image of the road in front of 
the vehicle. If the digital image pickup 11 is mounted at 
approximately the center front of the vehicle, the bottom 
center portion of the captured image will necessarily 
contain the road surface, as is described further below. 

The brightness histogram generator 12 produces a 
brightness histogram, as shown in Fig. 27, from the input 
road image data. Using this brightness histogram, the 
brightness histogram generator 12 also detects the ve- 
hicle position in the image, i.e., detects the luminance 
at the bottom middle part of the image. 

Using the histogram, the lane brightness separator 
13 detects the change in brightness to the right and left 
sides of the luminance at the vehicle position In the im- 
age. By detecting valleys in the brightness level, the lane 
brightness separator 13 finds the luminance range cor- 
responding to the lane area, and extracts the pixels in 
that range as lane area candidate pixels. 

Referencing the extracted lane area candidate pix- 
els, the lane polygon detector 14 finds the edges of the 
corresponding pixel area to the right and left sides of the 
bottom scanning line of the image. It then detects the 
midpoint between these side edges, and then repeals 
the edge pixel detection process at another scanning 
line higher in the image. The result of this process is a 
polygon that is detected as the road area. 

This process is described more specifically below 
with reference to the flow chart in Fig. 28. 

The first step is to generate a brightness histogram 
from the luminance data captured by the digital imaging 
pickup 11 (step 81 ). 

Using the brightness bO at the bottom center of the 
image, the position in the histogram corresponding to 
the road surface directly below the vehicle is obtained 
(step S2). A valley on the left side is detected first. 
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Mean brightness values b^e^^l 



ancJ b^^3^2 are 
then obtained for the overlapping brightness regions 
- bi < b < bo and b© - 3bi/2 < b < bo - bi/2 where b is a 
brightness parameter and b^ is a constant brightness 
level (step S3). 

The equation b^^^^a = b^^^^2 - b^^^^l Is then cal- 
culated (step S4). and the sign of b^^ean^ is then deter- 
mined (step S5). A valley is detected by determining 
when the sign changes from negative to positive. 

If the sign has not changed from negative to posi- 
tive, bo = bo - (step S6), and the procedure loops 
back to step S3 (step S6). This process is repeated until 
a valley is detected to the left side. 

A similar process is then executed to detect a valley 
on the right. The first step (step 87) is to reinitialize bo 
to the value detected in S2, and then calculate b^g^n^ 
and b^ean4 for the regions bo < b < bo + b^ and bo + b/ 

b 4 - 



2 < b < bn + 



3b/2. 



The equation b^^^^b = 



b^e3n3 IS then obtained (step S8). 

A valley is detected by determining when the sign 
changes from negative to positive. If the sign has not 
changed from negative to positive, bo = bo + b^/2 (step 
SI 0): and the procedure loops back to step S7 (step S9). 
This process is repeated until a valley is detected to the 
right side. 

The pixels in the brightness region between the de- 
tected valleys are then detected as the lane area can- 
didate region of the image (step S11). 

Referencing the extracted lane area candidate pix- 
els, the edges of the pixel areas to the right and left sides 
of the bottom scanning line of the image are detected 
as shown in Fig. 29. It then detects the midpoint between 
these side edges, and then repeats the edge pixel de- 
tection process at another scanning line higher in the 
image. The result of this process is a polygon as shown 
in Fig. 29 detected as the road area. 

As described above, the road area extraction appa- 
ratus of the present embodiment detects an area con- 
taining the luminance value directly below the vehicle in 
the brightness histogram. Using the image area corre- 
sponding to this brightness value, the edges of the road 
area are then detected using a center-outward pixel 
comparison method. The road edges thus extracted are 
an accurate representation of the road area in that im- 
age. 

A first alternative of second embodiment of a road 
area extraction apparatus according to the present in- 
vention is described next below with reference to Figs. 
30 and 31 . This road area extraction apparatus compris- 
es a digital image pickup 21 , brightness distribution pal- 
tern detector 22, and lane image separator 23. 

The digital image pickup 21 is typically a video cam- 
era used for capturing an image of the road in front of 
the vehicle. If the digital image pickup 21 is mounted at 
approximately the center front of the vehicle, the bottom 
center R portion of the captured image will necessarily 
contain the road surface, as is described further below. 

The brightness distribution pattern detector 22 sam- 



10 



15 



20 



2$ 



30 



35 



40 



45 



SO 



55 



pies a defined region So from the bottom middle of the 
road image data as shown in Fig. 31. and determines 
the brightness distribution a in this area. 

The lane image separator 23 then divides the lane 
image and compares the brightness distributbn b of 
each image segment with the brightness distribution 
pattern a obtained by the brightness distribution pattern 
detector 22 as shown in Fig. 32. If the patterns match, 
the image area is determined to be a road area. 

The operation of this road area extraction apparatus 
is described below with reference to the flow chart in 
Fig. 33. 

The brightness distribution pattern of the defined re- 
gion So at the bottom middle of the road image data is 
obtained by calculating brightness mean m and distri- 
bution s in region So- The maximum logarithmic proba- 
bility 1 is then calculated as follows where n is the 
number of data points in the region Sq. 



a2) = -(n/2) log27ia2 - n/2 



(12) 



The road image is then separated into N x N (the 
number of data points in this region), the brightness dis- 
tribution pattern of an area Si different from region SO 
is obtained by calculating brightness mean \x\ and dis- 
tribution ai in region Si. The maximum logarithmic prob- 
ability ii is then calculated as follows where n is the 
number of data points in the region Si. 

If the maximum logarithmic probability li is 1-a ^ Ii 
< 1 +a (where a is a positive integer) in step S3, the area 
is determined to be a lane in step S4. For example, re- 
gion SI is determined to be a lane because the distri- 
bution patterns are similar although the brightness lev- 
els of patterns a and b in Fig. 32 differ. 

If the above equation is not true, the area is deter- 
mined not to be a lane at step S5. 

If i > N x N at step S6, the process ends. If not. i is 
incremented to i + 1 at step S7. and the procedure loops 
back to step S2. 

As described above, the road area extraction appa- 
ratus of the present embodiment detects an area with 
the same brightness pattern as the area directly below 
the vehicle in the brightness histogram to accurately ex- 
tract the lane area. The overall image can also be seg- 
mented based on a probability comparison with the SO 
of the brightness pattern. 

A second alternative of a road area extraction ap- 
paratus according to the second embodiment of the 
present invention is described next below with reference 
to Figs. 34 and 35. This road area extraction apparatus 
comprises a digital image pickup 31 , brightness histo- 
gram generator 32, and lane image extractor 33. 

The digital image pickup 31 is typically a video cam- 
era used for capturing an image of the road in front of 
the vehicle as described in the preceding aternatives. 

The brightness histogram generator 32 produces a 
brightness histogram Irom the input road image data. 
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Using this brightness histogram, the brightness histo- 
gram generator 32 also detects the vehicle position in 
the image, i.e.. detects the luminance at the bottom mid- 
dle part of the image. In this histogram, the number of 
pixels having a brightness level greater than the bright- s 
ness at the vehicle position and corresponding to the 
road area Is substantially constant. This pixel count is 
defined as pixel count g below. Note that this pixel count 
g also varies according to such factors as the camera 
height and view angle. io 

The lane image extractor 33 determines the bright- 
ness range containing pixels corresponding to the road. 
The pixels of the road image in the brightness range de- 
termined to be the lane area of the brightness histogram 
are extracted to obtain the lane area. is 

The operation of the above road area extraction ap- 
paratus is described below with reference to the flow 
chart in Fig. 36. 

The first step is to generate a brightness histogram 
from the luminance data captured by the digital Imaging 20 
pickup (step SI). The brightness histogram shown in 
Fig. 35 is exemplary of this histogram. 

Using the brightness bO at the bottom center of the 
image, the sum s of the frequency, or number of pixels 
observed is obtained (step S2). 2S 

Using the frequency a of pixels with a higher bright- 
ness level, the sum s of the frequency is obtained (step 
S3). 

The lane area Is determined to have been extracted 
if s is greater than the previously obtained pixel count g 30 
corresponding to the lane area. If s is less than g, the 
procedure loops back to step S5 and S3, and the fre- 
quency ai of brightness bO + Db is obtained. If s is great- 
er than g, the procedure advances to step S5 and S6. 
The brightness b1 and original brightness bO at the ve- 3S 
hide position are then used to extract the pixels of 
brightness br where bO < br < b1 . 

These pixels are then located on the road image to 
obtain the lane area (step S7). 

The brightness values of the road use the common 40 
tendency of a road to have a continuous brightness level 
that is lower than the brightness immediately before the 
vehicle. The brightness immediately below the vehicle 
is therefore obtained from a brightness histogram, the 
brightness values containing the frequency of the 45 
number of pixels in the lane area of the image, which is 
previously determined, is obtained from this brightness 
value before the vehicle, and the lane area can Ihus be 
accurately extracted. 

A third alternative of a road area extraction appara- so 
tus according to the second embodiment of the present 
invention is described next below with reference to Figs. 
37 to 39. This road area extraction apparatus comprises 
a digital image pickup 41, brightness histogram gener- 
ator 42, image separator 43, lane area clustering unit 5S 
44, and lane width detector 45. 

The digital image pickup 41 is typically a video cam- 
era used for capturing an image of the road in front of 



the vehicle as described in the preceding alternatives. 

The brightness histogram generator 42 produces a 
brightness histogram from the input road image data as 
shown in Fig. 38. Using this brightness histogram, the 
brightness histogram generator 42 also detects the ve- 
hicle position in the image, i.e., detects the luminance 
at the bottom middle part of the image. 

The image separator 43 then detects all valleys in 
from the brightness distribution, producing a segmented 
lane image as shown in Fig. 39. 

Based on the lane width information from the lane 
width detector 45, the lane area clustering unit 44 merg- 
es the segment below the vehicle with the first horizontal 
segment, and then with a segment in the vertical direc- 
tion. The lane width detector 45 obtains the lane width 
from the image height based on the lane area obtained 
from the lane area clustering unit 44. 

The road area extraction apparatus thus comprised 
operates as described below with reference to the flow 
chart in Figs. 40 and 41 . 

The first step is to generate a brightness histogram 
as shown In Fig. 38 from the luminance data captured 
by the digital imaging pickup (step SI ). 

Using the brightness bO at the bottom center of the 
image, the position in the histogram corresponding to 
the road surface directly below the vehicle is obtained 
(step S2). A valley on the left side is detected first. 

Mean brightness values b^^ean'' ^mean^ 
then obtained for the overlapping brightness regions bO 
- b1 < b < bO and bO - 3b1/2 < b < bO - b1/2 where b Is 
a brightness parameter and b^ is a constant brightness 
level (step S3). 

The equation b^eanS = b^^anS - b^^^g^l is then cal- 
culated (step 84). and the sign of b^ean^ 'S then deter- 
mined (step S5). A valley is detected by determining 
when the sign changes from negative to positive. 

If the sign has not changed from negative to posi- 
tive, bo = bo - bi/2 (step S6), and the procedure loops 
back to step S3 (step S6). This process Is repeated to 
the left side of the histogram (step 87), and the proce- 
dure then advances to step S8. 

A similar process is then executed to detect a valley 
on the right. 

The first step (step SB) is to reinitialize b© to the val- 
ue detected in S2. and then calculate b^^^^S and ^^^^^^,4 
for the regions bo < b < b© + b^ and bo + bi/2 < b < bo + 
3bi/2. The equation b^^3„b = b^^3„4 - b^^anS 's then 
obtained (step S9). 

A valley is detected by determining when the sign 
changes from negative to positive (step S10, Fig. 41 ). If 
the sign has not changed from negative to positive, bo 
= bo + b-,/2 (step S 1 1 ), and the procedure loops back to 
step S7 (step S11 ). This process is repeated until a val- 
ley Is detected to the right side. 

Whether valleys have been detected to the right 
side of the histogram is then determined (step S12). If 
not. steps S11 and S8 are repeated. If step S12 returns 
YES. the pixels in the brightness region between the de- 
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tected valleys are then detected as the lane area can- 
didate region of the image (step SI 3). Fig. 39 represents 
the input image separated into the areas between the 
detected valleys. 

The area directly below the vehicle is then detected 5 
(step SI 4). Region SO in Fig. 39 is this target region. 
Target region SO Is then combined with the region S^ 
adjacent thereto (step SI 5). 

The previously obtained lane width wf (h) is then ref- 
erenced to calculate horizontal width w(h) (step SI 6). io 

If wf(h) > w(h) (step SI 7), the procedure loops back 
to step SI 5. If wf < w (step SI 7) and the height h of the 
combined regions is greater than a defined height hO, 
the procedure terminates. 

Note that hO is the height of the vanishing point of is 
the road, which is a constant height determined by the 
installation height ot the digital image pickup 41 if the 
road is level. If h < hO. the combined region is combined 
with the region adjacent thereto above and narrower 
than the top width of the lower combined region; In Fig. 
39 regions S2 and S3 are combined. The procedure fi- 
nally terminates when the height h = hO. 

By thus segmenting the image based on the lane 
width, and using known properties of the road configu- 
ration in the image to extract the lane area, accurate 2S 
lane area extraction is possible. Note that the lane area 
can also be extracted by segmenting the image using a 
brightness pattern comparison in combination with the 
method described above. 

A fourth alternative of a road area extraction appa- 30 
ratus according to the second embodiment of the 
present invention is described next below with reference 
to Figs. 42 to 44. This road area extraction apparatus 
comprises a digital image pickup 51. brightness histo- 
gram generator 52, image separator 53, lane area clus- 3S 
tering unit 54. lane width detector 55, lane region corre- 
lation detector 56. and lane extending direction detector 
57. 

The digital image pickup 51 is typically a video cam- 
era used for capturing an image of the road in front of 40 
the vehicle as described in the preceding alternatives. 

The brightness histogram generator 52 produces a 
brightness histogram from the input road image data as 
shown in Fig. 38. Using this brightness histogram, the 
brightness histogram generator 52 also detects the ve- ^5 
hide position in the image, i.e.. detects the luminance 
at the bottom middle part of the image. 

The image separator 53 then delects all valleys in 
from the brightness distribution. 

Based on the lane width information from the lane so 
width detector 55= the lane area clustering unit 54 merg- 
es the segment below the vehicle with the first horizontal 
segment, and then with a segment in the vertical direc- 
tion. The lane width detector 55 obtains the lane width 
from the image height based on the lane area obtained 55 
from the lane area clustering unit 54. 

The lane region correlation detector 56 calculates 
the correlation between the lane area delected by the 
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lane area clustering unit 54. and a triangle defined by a 
particular point on the vanishing line and two lane edge 
positions at the bottom of the image supplied from the 
lane width detector 55. 

The lane extending direction detector 57 detects the 
direction of the lane relative to the vehicle from the point 
on the vanishing line providing the highest correlation 
value. 

The road area extraction apparatus thus comprised 
operates as described below with reference to the flow 
chart in Fig. 42. 

The lane width is first detected (step SI) using the 
method of the third alternative of the second embodi- 
ment described above. 

A point r on the vanishing line is then initialized to 
0. I.e.. to the coordinate value at the left edge of the van- 
ishing line (step 32). 

The correlation between the extracted lane area 
and the triangle formed using point r = 0, I.e. , the number 
of matching pixels, is then calculated. 

Point r is then incremented to r = r +1 (step S4), thus 
moving the vanishing point one pixel to the right. If r does 
not exceed pixel rh at the right edge of the image (step 
S5). the procedure loops back to step S3 to repeat the 
correlation calculation. If r is beyond the right edge, the 
value of r resulting in the highest correlation is detected 
(step S6). If r is greater than half (rh/2) the number of 
horizontal pixels in the image, the lane is to the right of 
the vehicle. If r < rh/2, the lane is to the left of the vehicle. 

By thus obtaining the correlation between the de- 
tected lane area and a triangle formed between the van- 
ishing point and the lane edges, the direction of the lane 
relative to the vehicle can be easily obtained using the 
maximum correlation value. 

A local positioning apparatus according to the first 
embodiment of the invention can thus separately detect 
the contour points of the road area and lane marker can- 
didates using low and high frequency Images with no 
spatial frequency overlap. The effects of noise can 
therefore be reduced when compared with lane extrac- 
tion using only a high frequency image, the method of 
the prior art, and lane markers can be detected with high 
precision. 

The lane area can also be accurately detected in 
the road image by means of the road area extraction 
apparatus according to the second embodiment of the 
invention by detecting the lane area from the image da- 
ta, and delecting a lane area polygon starting from the 
bottom center of the image, a point corresponding to di- 
rectly in front of the vehicle. 

The lane area can also be accurately detected in 
the road image by detecting Ihe area containing the lu- 
minance values of the pixels directly in front of the ve- 
hicle from a brightness histogram, and using the image 
area corresponding thereto to detect the edges of the 
lane area starting from the bottom center of the image, 
a point corresponding to directly in front of the vehicle. 

The lane area can also be accurately detected by 
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detecting patterns identical to the brightness distribution 
pattern drrectly before the vehicle. 

The lane area can also be accurately detected by 
detecting the area containing the luminance values of 
the pixels directly in front of the vehicle from a brightness s 
histogram, calculating the brightness values containing 
the frequency of the number of pixels in the lane area 
of the image, which is previously determined, and de- 
fining the pixels in this area as the lane area. 

The lane area can also be accurately detected by io 
calculating image segments by a road width and lumi- 
nance pattern comparison, and using known features of 
the road configuration in the image to extract the lane 
area. 

The invention being thus described, it will be obvi- 
ous that the same may be varied in many ways. Such 
variations are not to be regarded as a departure from 
the spirit and scope of the invention, and all such mod- 
ifications as would be obvious to one skilled in the art 
are intended to be included within the scope of the fol- 20 
lowing claims. 

Claims 

2S 

1. A local positioning apparatus (LP) for detecting a 
local position of a subject (AM) capable of advanc- 
ing in a direction relating to a lane in a local area 
based on a digital image signal (Si) representing a 
local image (Vi) of an area in an advancing direction 30 
of said subject (Af^), said apparatus comprising: 
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said lane detection means (500) compares said 
road contour data (Sre)and second road region 
data (Srr) to detect whether said second road 
region data (Srr) or lane candidate area (AeR 
and AeL) is larger 

3. A local positioning apparatus (LP) according to 
claim 2, wherein said lane detection means (500) 
defines said road contour data (Sre) as the lane de- 
tection signal (Sre) when said second road region 
data (Srr) is within said lane candidate area (AeR 
and AeL). 

4. A local positioning apparatus (LP) according to 
claim 2, wherein said lane detection means (500) 
causes said contour extraction means (400) to ex- 
tract a new contour when said second road region 
data (Srr) is not within said lane candidate area 
(AeR and AeL). 

5. A local positioning apparatus (LP) according to 
claim 1 , further comprising an in-lane position de- 
tector (600) for detecting said position of said sub- 
ject (AM) in said lane based on said lane detection 
signal (Sre). 

6. A local positioning apparatus (LP) according to 
claim 1 , wherein said lane area extraction means 
(300) extracts pixels in luminance signal (SL) in a 
particular luminance range (Bmin - Bmax) to pro- 
duce road region data (Srr). 



2. 



a first image signal generating means (204) for 7. 
extracting a high spatial frequency component 
from the digital image signal (Si), and producing 35 
an edge signal (SH); 

a contour extraction means (400) for extracting 
a contour of a lane based on the edge signal 
(SH). and generating road contour data (Sre): 
a second image signal generating means (202) "to 
for extracting a low spatial frequency compo- 
nent from the digital image signal (Si), and pro- 
ducing a luminance signal (SL); 

a lane area extraction means (300) for extract- 8. 

ing the lane area based on the luminance signal ^5 

(SL) and generating road region data (Srr); and 

a lane detection means (500) for detecting the 

lane position based on the lane contour data 

(Sre) and road region data (Srr). and producing 

a lane detection signal (Sre). so 

A local positioning apparatus (LP) according to 
claim 1, wherein said lane area extraction means 
(300) comprises a lane region setting means (306) 
for adding a particular tolerance range (Wa) to said ss 
road region data (Srr) to produce second road re- 9. 
gion data (Srr) representing a lane candidate area 
(AeR and AeL) in which a lane may be present; and 



A local positioning apparatus (LP) according to 
claim 6, wherein said lane area extraction means 
(300) comprises a first pixel luminance extraction 
means (302) for extracting a first luminance (Bmax) 
from a pixel (Pf) representing a farthest point re- 
moved a first particular distance (yPf); and 

extracts pixels with luminance below said first 
luminance (Bmax) to generate road region data 
(Srr). 

A local positioning apparatus (LP) according to 
claim 7. wherein said lane area extraction means 
(300) further comprises a second pixel luminance 
extraction means (302) for extracting a second lu- 
minance (Bmin) from a pixel (Pn) representing a 
point closer than first particular distance (yPf); and 

extracts pixels with luminance between said 
first luminance (Bmax) and said second lumi- 
nance (Bmin) to generate road region data 
(Srr). 

A local positioning apparatus (LP) according to 
claim 7. wherein first particular distance (yPf) is set 
according to the speed o\ travel (Sv) of said subject 
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(AM). 

10. A. local positioning apparatus (LP) according to 
claim 7, wherein first pixel luminance extraction 
means (302) extracts a first luminance (Bmax) from s 
a pixel (Pf) representing an image at a particular 
horizontal position (xPf) on a horizontal line passing 
through a point removed a first particular distance 
(yPf). 

10 

11. A local positioning apparatus (LP) according to 
claim 10, wherein said particular horizontal position 
(xPf) is determined ad hoc according to said steer- 
ing angle (Ss) of said subject (AM). 

75 

12. A local positioning method for detecting a local po- 
sition of a subject (AM) capable of advancing in a 
direction relating to a lane in a local area based on 
a digital image signal (Si) representing a local im- 
age (Vi) of an area in an advancing direction of said 20 
subject (AM): comprising. 

a first image signal generating step for extract- 
ing a high spatial frequency component from 
said digital image signal (Si), and producing an 2S 
edge signal (SH); 

a contour extraction step for extracting a con- 
tour of a lane based on said edge signal (SH), 
and generating road contour data (Sre); 
a second image signal generating step for ex- 30 
tracting a low spatial frequency component 
from said digital image signal (Si), and produc- 
ing a luminance signal (SL): 
a lane area extraction step for extracting said 
lane area based on said luminance signal (SL) 35 
and generating road region data (Srr); and 
a lane detection step for detecting said lane po- 
sition based on said lane contour data (Sre) and 
road region data (Srr), and producing a lane de- 
tection signal (Sre). 40 

1 3. A road area detection method for extracting a road 
area as a polygonal area in an image plane cen- 
tered on a vehicle, comprising: 

45 

a road area extraction method for segmenting 
a road area using road image data; and 
a road polygon detection method for a polygo- 
nal area including a point directly before a ve- 
hicle in an image plane from said areas sepa- so 
rated by said road area extraction method. 

14. A road area extraction apparatus comprising: 

a road imaging device for capturing an image 55 
of a road before a vehicle; 
a brightness histogram generating means for 
creating a brightness histogram using said road 
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image data from said road imaging device; 
a road brightness separating means for sepa- 
rating from a brightness histogram said area 
containing luminance values immediately be- 
fore said vehicle and contained between val- 
leys in said brightness histogram generated by 
said brightness histogram generating means; 
and 

a road polygon detecting means for detecting 
a polygonal area including a point directly be- 
fore a vehicle in an image plane in said areas 
separated by said road brightness separating 
means; 

wherein said road area is extracted as a polyg- 
onal area in an image plane centered on a ve- 
hicle. 

15. A road area extraction apparatus comprising: 

a road imaging device for capturing an image 
of a road before a vehicle; 
a brightness pattern detection means for de- 
tecting said surface brightness pattern immedi- 
ately before said vehicle using said road image 
data from said road imaging device; and 
a brightness pattern separation means for sep- 
arating road areas with a brightness pattern 
similar to said road brightness pattern immedi- 
ately before said vehicle supplied from said 
brightness pattern detection means, wherein a 
road area is separated from said image by 
measuring said brightness pattern of said road 
surface, 

16. A road area extraction apparatus comprising: 

a road imaging device for capturing an image 
of a road before a vehicle; 
a brightness histogram generating means for 
creating a brightness histogram using said road 
image data from said road imaging device; and 
a pixel count road area extractbn means for 
setting on said brightness histogram a range of 
a predetermined pixel count constituting a road 
area using a brightness position directly before 
a vehicle using said brightness histogram gen- 
erated by said brightness histogram generating 
means, wherein a brightness area is set in said 
brightness histogram using a brightness value 
immediately before said vehicle by first obtain- 
ing said pixel count of said road area to sepa- 
rate road areas in said image. 

17. A road area extraction apparatus comprising: 

a road imaging device for capturing an image 

of a road before a vehicle; 

an image separating means for segmenting an 
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image from road image data from said road im- 
aging device based on image characteristics; 
a road area combining means for combining a 
segmented area of said vehicle position in said 
image with areas adjacent thereto on both s 
sides for said road width only using road width 
data from a road width detection means de- 
scribed below and segment data from said im- 
age separating means, and repeating this op- 
eration to a particular position toward said top io 
of said image to extract a road area; and 
a road width detection means for calculating 
said road width at various heights in said image 
from said extracted road areas from said road 
area combining means, wherein image seg- ^5 
ments are combined to extract said road area 
based on road width information detected from 
said previous image. 

1 8. A road area extractbn apparatus according to claim 20 
17, wherein said road area combining means com- 
bines segmented areas at said vehicle position in 
said lateral direction to said road width, and com- 
bines areas vertically such that said upper area has 

a width less than said width of said lower area. 2S 

19. A road direction detection apparatus comprising: 

a road imaging device for capturing an image 
of a road before a vehicle; 30 
an image separating means for segmenting an 
image from road image data from said road im- 
aging device based on image characteristics; 
a road area combining means for combining a 
segmented area of said vehicle position in said 3S 
image with areas adjacent thereto on both 
sides for said road width only using road width 
data from a road width detection means de- 
scribed below and segment data from said im- 
age separating means, and repeating this op- 40 
eration to a particular position toward said top 
of said image to extract a road area; 
a road width detection means for calculating 
said road width at various heights in said image 
from said extracted road areas from said road 45 
area combining means; 

a road area correlation detection means for cal- 
culating a correlation between a triangle de- 
fined by a vanishing point at a selected position 
on a particular vanishing line and two road edge so 
data points at said image bottom determined by 
said road width detection means, and an ex- 
tracted road area supplied by said road area 
combining means, and detecting said vanish- 
ing point yielding said greatest correlation val- 55 
ue; and 

a road direction detection means for detecting 
said direction of said road to said front of said 



vehicle from said vanishing point position yield- 
ing said greatest correlation value as detected 
by said road area correlation detection means, 
wherein the direction of said road relative to the 
vehicle is detected by obtaining said correlation 
between an extracted road area and a triangle 
defined by a vanishing point at a selected po- 
sition on a particular vanishing line and two 
road edge data points. 
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